Simulation of tangible user interface interactions and gestures using array of haptic cells

ABSTRACT

A user interface device includes a flexible layer comprising a touch surface configured to receive a touch by a user, a plurality of haptic cells covered by the flexible layer, each haptic cell comprising a haptic output device, a sensor configured to sense an amount and/or rate of deformation of the flexible layer when a user touches the touch surface, and a processor configured to receive an output signal from the sensor, generate a haptic control signal based on the output signal from the sensor, and output the haptic control signal to at least one haptic output device of the plurality of haptic cells to cause the haptic output device to deform an associated haptic cell in response to the sensed deformation of the flexible layer.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 14/262,482, filed Apr. 25, 2014, which claims thebenefit of priority to U.S. Provisional Patent Application No.61/816,605, filed Apr. 26, 2013, the entire contents of which areincorporated herein by reference.

FIELD

The present invention is directed to the simulation of tangible userinterface interactions and gestures using an array of haptic cells.

BACKGROUND

Current user interfaces in the form of touchscreens use audio, video,and in some cases vibrotactile haptics, to display digital informationto a user. Current implementations of touchscreens also typicallyrequire constant visual attention for interaction. Although tangible(i.e. physical) user interface elements can facilitate user interfaceinteractions, particularly in visually occupied scenarios, such asdriving, user interface interactions and gestures that are currentlyimplemented in digital devices typically lack in physical and realismaspects.

It is desirable to increase the range of applications of hapticallyenabled touchscreens, such as increasing the perceived resolution of thetouchscreen and expanding the breadth and depth of the hapticinformation that can be delivered by such touchscreens. It is alsodesirable to enhance the fidelity and realism of user interfaceinteractions and gestures through physical simulation of the userinterface elements to create tangible and efficientinteractions/interfaces, and to improve the user experience.

SUMMARY

According to an aspect of the present invention, therein is provided auser interface device that includes a flexible layer comprising a touchsurface configured to receive a touch by a user, a plurality of hapticcells covered by the flexible layer, each haptic cell comprising ahaptic output device, a sensor configured to sense an amount and/or rateof deformation of the flexible layer when a user touches the touchsurface, and a processor configured to receive an output signal from thesensor, generate a haptic control signal based on the output signal fromthe sensor, and output the haptic control signal to at least one hapticoutput device of the plurality of haptic cells to cause the hapticoutput device to deform an associated haptic cell in response to thesensed deformation of the flexible layer.

In an embodiment, the processor is configured to output a plurality ofhaptic control signals to a plurality of haptic output devices locatedin proximity to one another to collectively create a haptic effect.

In an embodiment, the haptic effect simulates a button or a joystick.

In an embodiment, at least two of the plurality of haptic output devicescreate different amounts of deformations of their associated hapticcells to create the haptic effect.

In an embodiment, the processor is configured to generate a plurality ofhaptic control signals based on the output signal from the sensor andoutput the haptic control signals to a plurality of haptic outputdevices sequentially to create a haptic effect.

In an embodiment, the haptic effect simulates a wave or a ripple. In anembodiment, the haptic effect simulates movement of a joystick or aslider.

In an embodiment, the processor is configured to generate a secondhaptic control signal different than the haptic control signal andoutput the second haptic control signal to the haptic output device tocause the haptic output device to output a vibrotactile haptic effect.

In an embodiment, the user interface device further includes a pressuresensor configured to sense a resistance displayed by the haptic cell tothe user at the touch surface. In an embodiment, the processor isconfigured to change the haptic control signal based on the sensedresistance displayed by the haptic cell.

According to an aspect of the present invention, there is provided amethod that includes sensing an amount and/or rate of displacement of aflexible layer of a user interface device, generating a haptic controlsignal based on the sensed amount and/or rate of displacement of theflexible layer with a processor, and deforming a haptic cell with ahaptic output device based on the haptic control signal.

In an embodiment, the method includes generating a plurality of hapticcontrol signals and deforming a plurality of haptic cells located inproximity to each other with a plurality of haptic output devices basedon the plurality of haptic control signals to collectively create ahaptic effect. In an embodiment, at least two of the haptic cells aredeformed by different amounts to create the haptic effect.

In an embodiment, the method includes generating a plurality of hapticcontrol signals and sequentially deforming a plurality of haptic cellslocated in proximity to each other with a plurality of haptic outputdevices based on the plurality of haptic control signals to collectivelycreate a haptic effect.

In an embodiment, the method includes generating a second haptic controlsignal different than the haptic control signal and generating avibrotactile haptic effect with the haptic output device.

In an embodiment, the method includes sensing a resistance displayed bythe haptic cell to a user of the user interface device. In anembodiment, the method includes changing the haptic control signal basedon the sensed resistance displayed by the haptic cell.

These and other aspects, features, and characteristics of the presentinvention, as well as the methods of operation and functions of therelated elements of structure and the combination of parts and economiesof manufacture, will become more apparent upon consideration of thefollowing description and the appended claims with reference to theaccompanying drawings, all of which form a part of this specification.It is to be expressly understood, however, that the drawings are for thepurpose of illustration and description only and are not intended as adefinition of the limits of the invention. As used in the specificationand in the claims, the singular form of “a”, “an”, and “the” includeplural referents unless the context clearly dictates otherwise.

BRIEF DESCRIPTION OF THE DRAWINGS

The components of the following Figures are illustrated to emphasize thegeneral principles of the present disclosure and are not necessarilydrawn to scale. Reference characters designating correspondingcomponents are repeated as necessary throughout the Figures for the sakeof consistency and clarity.

FIG. 1 is a schematic illustration of a system according to embodimentsof the invention;

FIG. 2 is a schematic exploded view of an embodiment of the system ofFIG. 1 in the form of a user interface;

FIG. 3 is a schematic top view of the user interface of FIG. 2;

FIG. 4A is a schematic side view of a haptic cell of the user interfaceof FIG. 2 in a first state;

FIG. 4B is a schematic side view of the haptic cell of the userinterface of FIG. 4A in a second, deformed state;

FIG. 5 is an embodiment of a processor of the system of FIG. 1;

FIG. 6 is a schematic side view of an embodiment of the haptic cell ofthe user interface device of FIG. 2;

FIG. 7 is an illustration of voltage as a function of time for twodifferent haptic control signals for driving a haptic output device ofthe haptic cell of FIG. 4A; and

FIG. 8 is an illustration of voltage as a function of time for a singlehaptic control signal for driving the haptic output device of the hapticcell of FIG. 4A.

DETAILED DESCRIPTION

FIG. 1 is a schematic illustration of a system 100 in accordance with anembodiment of the invention. As illustrated, the system 100 includes aprocessor 110, a memory device 120, and input/output devices 130, whichare interconnected via a bus 140. In an embodiment, the input/outputdevices 130 may include a touchscreen device 150, a haptic output device160 and/or other input devices that receive input from a user of thesystem 100 and output devices that output information to the user of thesystem 100. In an embodiment, the system 100 may be a user interface inthe form of a touch mobile or tablet device that includes all of thecomponents illustrated in FIG. 1 in a single integrated device.

The touchscreen device 150 may be configured as any suitable userinterface or touch/contact surface assembly and may be configured forphysical interaction with a user-controlled device, such as a stylus,finger, etc. In some embodiments, the touchscreen device 150 may includeat least one output device and at least one input device. For example,the touchscreen device 150 may include a visual display 152 configuredto display, for example, images and a touch sensitive screen comprisingat least one sensor 154 superimposed thereon to receive inputs from auser's finger or stylus controlled by the user. The visual display 152may include a high definition display screen.

The sensor 154 may include a strain gauge sensor to measure thedeformation magnitude during interactions between the user and thetouchscreen device 150, a force-sensitive resistor (“FSR”) sensor tomeasure the force/stress applied to the touchscreen device 150, amulti-touch touch sensor to detect the location of touch input(s) in theuser's single or multiple touch interactions, and/or a multi-touchpressure sensor to measure the pressure applied under each touchlocation. In some embodiments, the sensor 154 may include a temperatureor humidity or atmospheric pressure sensor to capture environmentalconditions, an accelerometer or gyroscope or magnetometer tocharacterize the motion, velocity, acceleration and/or orientation ofthe device, or a microphone to capture the user's voice command orenvironmental audio information. In addition, the sensor 154 may includea wireless transmitter to receive or transmit information from/to otherdevices wirelessly.

In various embodiments, the haptic output device 160 is configured toprovide haptic feedback to the user of the system 100 while the user isin contact with a least a portion of the system 100. For example, thehaptic output device 160 may provide haptic feedback to the touchscreendevice 150 itself to impose a haptic effect when the user is in contactwith the touchscreen device 150 and/or to another part of the system100, such as a housing containing at least the input/output devices 130.As discussed in further detail below, the haptic effects may be used toenhance the user experience when interacting with the system 100.

The haptic feedback provided by the haptic output device 160 may becreated with any of the methods of creating haptic effects, such asdeformation, kinesthetic sensations, vibration, electrostatic orultrasonic friction, etc. In an embodiment, the haptic output device 160may include non-mechanical or non-vibratory devices such as those thatuse electrostatic friction (“ESF”), ultrasonic surface friction (“USF”),or those that induce acoustic radiation pressure with an ultrasonichaptic transducer, or those that use a haptic substrate and a flexibleor deformable surface, or those that provide thermal effects, or thosethat provide projected haptic output such as a puff of air using an airjet, and so on. The haptic output device 160 may include an actuator,for example, an electromagnetic actuator such as an Eccentric RotatingMass (“ERM”) in which an eccentric mass is moved by a motor, a LinearResonant Actuator (“LRA”) in which a mass attached to a spring is drivenback and forth, or a “smart material” such as piezoelectric materials,electro-active polymers or shape memory alloys, a macro-composite fiber(“MCF”) actuator, an electro-static actuator, an electro-tactileactuator, and/or another type of actuator that provides a physicalfeedback such as vibrotactile feedback. Multiple haptic output devices160 may be used to generate different haptic effects.

The processor 110 may be a general-purpose or specific-purpose processoror microcontroller for managing or controlling the operations andfunctions of the system 100. For example, the processor 110 may bespecifically designed as an application-specific integrated circuit(“ASIC”) to control output signals to the haptic output device 160 toprovide haptic effects. The processor 110 may be configured to decide,based on predefined factors, what haptic effects are to be generatedbased on a haptic control signal received or determined by the processor110, the order in which the haptic effects are generated, and themagnitude, frequency, duration, and/or other parameters of the hapticeffects. The processor 110 may also be configured to provide streamingcommands that can be used to drive the haptic output device 160 forproviding a particular haptic effect. In some embodiments, the processor110 may actually include a plurality of processors, each configured toperform certain functions within the system 100. The processor 110 isdescribed in further detail below.

The memory device 120 may include one or more internally fixed storageunits, removable storage units, and/or remotely accessible storageunits. The various storage units may include any combination of volatilememory and non-volatile memory. The storage units may be configured tostore any combination of information, data, instructions, software code,etc. More particularly, the storage units may include haptic effectprofiles, instructions for how the haptic output device 160 is to bedriven, or other information for generating haptic effects.

FIG. 2 illustrates an embodiment of the system 100 of FIG. 1 in the formof a user interface device 200. As illustrated, the user interfacedevice 200 includes a touchscreen device 210, a display 220, and ahousing 230. In an embodiment, the touchscreen device 210 may be made ofsubstantially transparent materials so that images output by the display220 may be seen through the touchscreen device 230. The display 220 maybe any type of display such as a cathode ray tube (“CRT”), liquidcrystal display (“LCD”), light emitting diode (“LED”) display, plasmadisplay, flat panel display, flexible display or the like. Thetouchscreen device 210 and the display 220 may be installed togetherwithin the housing 230. In an embodiment, the touchscreen device 210 andthe display 220 may be integrated into the same unit or device. In anembodiment, the user interface device 200 may not include the display220. In an embodiment, the user interface device 200 may be flexible,e.g. permanently conformed, bendable, foldable, or rollable.

In an embodiment, the touchscreen device 210 includes a flexible layer402 (illustrated in FIG. 4A) and a plurality of haptic cells 212arranged in an array or grid. The haptic cells 212 may be separated byborders 214, as illustrated in FIG. 3, which shows a schematic top viewof the touchscreen device 210. In an embodiment, the haptic cells 212may not be separated by borders and instead abut against each other.Each of the haptic cells 212 is configured to provide a haptic effect inresponse to an input independent of the other haptic cells 212. Forexample, when multiple contacts are depressed on the touchscreen device210 substantially simultaneously, the haptic cells 212 may be configuredto generate multiple haptic effects in response to the multiplecontacts. The multiple contacts may be made by one finger or multiplefingers. The dimension or size of each of the haptic cells 212 may beconfigured to be less than 5 mm×5 mm, or less than 4 mm×4 mm, or lessthan 3 mm×3 mm, or less than 2 mm×2 mm, or less than 1 mm×1 mm.

FIG. 4A illustrates a schematic side view of an embodiment of one of thehaptic cells 212. The haptic cell 212 includes a haptic output device410, and electrical connectors 411 that connect the haptic output device410 to a power source and the processor 110 described above. Theflexible layer 402 has a touch surface 403 configured to receive inputsfrom the user of the device and overlays the haptic output device 410,as illustrated. In an embodiment, the flexible layer 402 may overlay theentire array of haptic cells 212. In an embodiment, each haptic cell 212may include a flexible layer 402 that is separate from the flexiblelayers 402 of adjacent haptic cells 212. In an embodiment, at least oneof the sensors 154 described above may be embedded in or otherwisecoupled to the flexible layer 402 such that the sensor 154 can detect atouch input at the touch surface 403 by the user.

The haptic output device 410 may include any type of haptic outputdevice that may cause a deformation of the flexible layer 402 whenenergized by a power source, for example, any of the devices describedabove with respect to the haptic output device 160 of FIG. 1. Forexample, the haptic output device may include a micro-fluidic display, apiezoelectric material or a composite piezoelectric material, anelectro-active polymer (“EAP”), a shape memory alloy, amicro-electro-mechanical system (“MEMS”) device, which may include aMEMS pump, smart gels, electro/magneto-rheological fluids, a thermalfluid pocket, a resonant device, a variable porosity membrane, a laminarflow modulation device, and/or an electromagnetic actuator.

When the processor 110 outputs a haptic control signal to the hapticoutput device 410, as described in further detail below, the shape ofthe haptic output device 410 and therefore the haptic cell 212 maydeform, as illustrated in FIG. 4B. The array of haptic cells 212 may beintegrated into the touchscreen of mobile and tablet devices, with eachhaptic cell 212 configured to display (i.e. output) haptic effects overa wide range of frequencies, from quasi-static deformation to highfrequency vibration. The mechanical assembly configurations andarrangement of the haptic cells 212 may allow for accumulating the forceand displacement of adjacent haptic cells to create programmable smoothrelieved (protrusion) or recessed (indentation) features, i.e., freeform deformation, as described in further detail below.

FIG. 5 illustrates an embodiment of the processor 110 in more detail.The processor 110 may be configured to execute one or more computerprogram modules. The one or more computer program modules may includeone or more of a sensor module 112, a determination module 114, a hapticoutput device control module 116, and/or other modules. The processor110 may also include electronic storage 118, which may be the same asthe memory device 120 or in addition to the memory device 120. Theprocessor 110 may be configured to execute the modules 112, 114, and/or116 by software, hardware, firmware, some combination of software,hardware, and/or firmware, and/or other mechanisms for configuringprocessing capabilities on processor 110.

It should be appreciated that although modules 112, 114, and 116 areillustrated in FIG. 2 as being co-located within a single processingunit, in embodiments in which the processor 110 includes multipleprocessing units, one or more of modules 112, 114, and/or 116 may belocated remotely from the other modules. The description of thefunctionality provided by the different modules 112, 114, and/or 116described below is for illustrative purposes, and is not intended to belimiting, as any of the modules 112, 114, and/or 116 may provide more orless functionality than is described. For example, one or more of themodules 112, 114, and/or 116 may be eliminated, and some or all of itsfunctionality may be provided by other ones of the modules 112, 114,and/or 116. As another example, the processor 110 may be configured toexecute one or more additional modules that may perform some or all ofthe functionality attributed below to one of the modules 112, 114,and/or 116.

The sensor module 112 is configured to receive an input signal from thesensor 154 that is generated when the sensor 154 detects an input from auser of the system 100 or user interface device 200. In embodiments inwhich there are multiple sensors, the sensor module 112 is configured toreceive and process input signals from the multiple sensors. The sensormodule 112 may be configured to determine whether the sensed input is anintentional input or merely an inadvertent touch to the touchscreendevice 150, 210 by comparing the strength of the input signal to apredetermined threshold strength that corresponds to an intentionalinput. The sensor module 112 is also configured to send a signal to thedetermination module 114 for further processing.

The determination module 114 is configured to determine what wasintended by the user when providing an input to the sensor 154. Forexample, the user may touch a certain location of the touchscreen device150, 210 or provide a particular gesture to the touchscreen device 150,210 that indicates that a certain function is to be performed by thesystem 100. The determination module 114 may be programmed with alibrary of predetermined gestures and touch locations on the touchscreendevice 150, 210 so that when the user touches a particular location onthe touchscreen device 150, 210 or provides a gesture to the touchscreendevice 150, 210, the determination module 114 may determine acorresponding output. In addition, the determination module 114 may alsooutput a signal to the haptic output device control module 116 so that ahaptic effect in accordance with embodiments of the invention describedbelow may be provided to the user.

The haptic output device control module 116 is configured to receive theoutput signal from the determination module 114 and determine the hapticeffect to be generated by the haptic output device 160, based on thesignal generated by the determination module 114. Determining the hapticeffect may include determining the type of haptic effect and one or moreparameters of the haptic effect, such as amplitude, frequency, duration,etc., of the haptic effect that will augment, for example, a tangibleuser interface element, such as a control button, as discussed infurther detail below.

In an embodiment, the user interface 200 may include an impedancedisplay 600, a portion of which is illustrated in FIG. 6, that uses acombination of pressure sensing, deformation/displacement sensing,deformation rate/displacement rate, and/or deformation displayfunctionalities of the array of haptic cells 212 to simulate themechanical impedance/stiffness of an object or medium in the virtualenvironments being displayed by the display 220 of the touchscreendevice 210. In an embodiment, the impedance may be rendered throughforce control schemes that close the loop with position feedback, asdescribed in further detail below. Implementations of such an embodimentprovide the ability to display hard vs. soft object interactions ingaming applications, augmented reality applications, medicalapplications, handshake communications, etc.

The mechanical properties of most physical objects, in terms of theresistance displayed against deformation, may be characterized usingstiffness (k) and damping (λ) parameters. For example, in an embodimentof the invention, the interaction of a user's finger may be described interms of the amount of deformation of the flexible layer 402 incurred(represented by “x”) and the rate of the deformation (represented by“v”), at each moment in time. In this embodiment, the amount ofdeformation (x) and/or the rate of deformation (v) may be measured by adisplacement sensor 602, such as a strain gauge, etc., embedded in orotherwise coupled to the flexible layer 402. The resistance beingdisplayed to the user (represented by “Fr”) may be measured using apressure sensor 604 embedded in or otherwise coupled to the flexiblelayer 402. To simulate a certain material, the desired resistance(represented by “F”) displayed against the user's movement should followequation (1):

F=k*x+λ*v  (1)

In this embodiment, the processor 110 may be programmed so that theresistance displayed to the user (Fr) is substantially equal to thedesired resistance (F) displayed against the user's movement inaccordance with equation (1). For example, if the value of F is lessthan the value of Fr, the processor 110 may output a haptic controlsignal via the haptic output device control module 116 to the hapticoutput device 410 to increase the resistance of the flexible layer 402,for example by applying a higher voltage to the haptic output device410, until F=Fr. Similarly, if the value of F is greater than the valueof Fr, the processor 110 may output a haptic control signal to thehaptic output device 410 to decrease the resistance of the flexiblelayer, for example by applying a lower voltage to the haptic outputdevice 410, until F=Fr. The constant measuring of Fr by the pressuresensor and the amount of displacement (x) and/or rate of displacement(v) by the displacement sensor, as well as calculating the value of Fmay be completed in a closed control loop executed by the determinationmodule 114 of the processor 110. In an embodiment, the values of k and λmay be constant and based on the desired feel of the resistance. In anembodiment, the values of k and λ may be a function of the amount ofdeformation (x) and/or deformation rate (v) at each moment in time.

For example, in an embodiment, it may be desirable to simulate the feelof honey if the display 220 is displaying an image of honey. Because ofthe viscoelastic properties of honey, it is desirable to output a higheramount of resistance if the rate of deformation is relatively high and alower amount of resistance if the rate of deformation is relatively low,which would give the user the feel as if he/she was pressing againsthoney. The processor 110 may be programmed with stiffness (k) anddamping (λ) coefficients as a function of deformation rate (v) so that amore realistic effect may be experienced by the user. As anotherexample, the processor 110 may be programmed to simulate a change inphysical condition of an object being displayed by the display 220 asthe user presses against the touch surface 403. For example, if the itembeing displayed is brittle, when the user begins to press against thetouch surface 403, the resistance being displayed to the user may berelatively high to simulate a hard object, but when the object “breaks”,the resistance may be sharply decreased to simulate the breaking of theobject.

In an embodiment, the touch surface 403 of the flexible layer 402 maysimulate a physically realistic user interface feature, such as a buttonor a joystick, which emerges from the touch surface 403, as illustratedin FIG. 4B. The user may move the joystick in a direction tangent to thetouch surface. In this embodiment, the processor 110 may be programmedto display programmable kinesthetic haptics via the haptic output device410 in the form of a resistive force against the user's motion parallelto the touch surface 403. Similar to the embodiment described above, apredetermined stiffness (k) and damping (λ) may be associated withmoving the joystick. The amount of deformation (x) and/or the rate ofdeformation (v) may be measured using the embedded displacement sensor602, and the resistance displayed against the user (Fr) may be measuredusing the pressure sensor 604. If the measured force displayed againstthe user (Fr) is smaller than the desired force F, as calculated usingequation (1) above, to be displayed against the user, the processor mayincrease the resistance by, for example, applying a higher voltage tothe haptic output device 410 to match F with Fr, and vice-versa, asdescribed above.

In some embodiments, the user may touch the touch surface 403 at alocation that is not directly over a single haptic cell 212 but insteadis in between haptic cells 212, such as on the boarder 214, or is over aplurality of haptic cells 212, depending on the size of the haptic cells212. In such situations, the processor 110 may use appropriateinterpolation techniques to replicate a sensation that the user isdirectly over a single haptic cell 212. For example, when the user'sfinger is placed between two haptic cells 212, the haptic rendering mayconsist of partially actuating the haptic output devices 410 of bothcells 212 in order to replicate the sensation that the user is on asingle cell 212. Implementations of this embodiment may provide the userwith an increased perceived haptic resolution. In addition,implementations of this embodiment may enable between-cell transitioneffects for information display at the sub-finger level, customizeddeformation feature size, such as for displaying skin stretch effects,etc.

In an embodiment, the processor 110 may be programmed to control thehaptic output devices 410 of adjacent haptic cells 212 in a sequence sothat a sense of movement or flow may be displayed and a sensationsimilar to a physical wave may be generated. For example, if the usertouches the touch surface 403 and slides his/her finger across the touchsurface 403, the processor 110 may generate haptic control signals inresponse to the sensed touches so that the haptic output devices 410 ofa plurality of haptic cells 212 are sequentially actuated from alocation corresponding to the initial touch to a location correspondingto the end of the touch so that the user will feel as if an object iscontinuously moving across the touch surface. In an embodiment, theshape of the wave and the travelling pace of the wave may be a functionof the properties of the material that is being simulated by thetouchscreen device 210. In an embodiment, ripples may be created byhaving multiple haptic output devices 410 located circumferentiallyaround the location of a touch input move in sequence away from thelocation of the touch. Such sequential activation of multiple hapticoutput devices 410 may be used to create flow-like motion, travelingwaves, and/or ripples, which may augment rotation and translation motionon the display 220, such as creating a vortex or a ripple shape, suchthat the center may be identified as the source of an event ornotification.

In an embodiment, the processor 110 may be configured to control thearray of haptic cells 212 with superimposed haptic control signals in away that the flexible layer 402 may kinaesthetically deform at specificlocations, and at the same time provide vibrotactile stimulus. Forexample, in an embodiment, the processor 110 may output a haptic controlsignal to the haptic output device 410 of at least one haptic cell 212to generate a button via kinaesthetic deformation, and then when thesensor senses the user has pressed the location of the flexible layer402 corresponding to the generated button, the processor 110 may outputanother haptic control signal to the haptic output device 410 togenerate and a click confirmation via a vibro-tactile effect.

In an embodiment, the haptic control signal output by the processor 110may consist of two components with different characteristics. First, asillustrated in FIG. 7, a low frequency and large magnitude sinusoid 710may be generated, which if played by itself may create a low frequencydeformation in each haptic cell 212 so that the haptic cell 212 may moveup and down in the center. Second, a high frequency and low magnitudesignal 720 may be generated, which if played alone may result in highfrequency vibrations in the haptic cell 212 and may be perceived asvibrotactile feedback. If the haptic output signal is a combination ofthe low frequency and large magnitude sinusoid 710 and the highfrequency and low magnitude signal 720, illustrated as signal 810 inFIG. 8, the haptic cell 212 may display kinesthetic haptic deformationand, simultaneously, vibrotactile haptics.

In an embodiment, the plurality of haptic cells 212 may be used togenerate isolated vibration effects for multi-output applicationsresponding to multiple inputs. In an embodiment, the plurality of hapticcells 212 may be used to simulate a guitar application in which both themacro deformations and the high frequency (micro-level) vibrationsassociated with the strings may be displayed.

In an embodiment, the array of haptic cells 212 may be used tophysically simulate a tangible touchscreen keyboard and physicallyaugment the text entry/keyboard interactions in touchscreen mobile andtablet devices. In other words, the array of haptic cells 212 may beused to physically display each key or the edges between the keys of akeyboard in a text entry application. Other tangible features may alsobe simulated to augment the key display. For example, the F and J keysmay be haptically marked, as they are in some physical keyboards, tofacilitate typing, or upon holding the modifier keys (e.g., CTRL),certain keys with predefined functions may be haptically highlighted(e.g., CTRL+C, CTRL+B, CTRL+V, etc.). The travel stroke and the forceprofile associated with each key may be tuned and adjusted so that ithas the geometry, travel, and force profile identical to that of a realbutton to enhance the fidelity and accuracy of the text entry/typingexperience, as compared to interaction with a real keyboard. Inaddition, the keys may be shaped to make the keyboard more ergonomic,such as rounded towards thumbs, etc.

In an embodiment, the array of haptic cells 212 may be used to enhancethe interaction of the user with a list of items (e.g., list scrolling)by raising or recessing items in a scrolled or static list, therebyfacilitating selections and increasing realism. For example, each itemin a static list may be raised so that the user's fingertip feels thetransitions as the fingertip slides from one to the next. In addition,marked items, such as “favorites” may have a different shape, texture,or vibrotactile feedback, and transitions between groups of items (e.g.,categories, alphabetic groups) may also be highlighted with a distinctfeature, such as a raised or recessed line. Similar deformations may beapplied to a scrolled list such that the shape would be felt by lightlytouching the list and feeling items slide by.

In an embodiment, the array of haptic cells 212 may be used tophysically augment and thus facilitate manipulating objects in a virtualenvironment, as it allows the user to physically interact with aprotrusion or recession “tied” to the object. Such a capability mayrender the interaction more realistic because it provides a perceptionthat is similar to real world object interaction scenarios. In addition,the force profile associated with such interactions may be programmable,as described above, which may allow for a richer haptically enabled userinterface that is capable of displaying a wide range of data about theobject and its properties. Examples of user interactions that maybenefit from this embodiment include, but are not limited tomoving/pushing the protrusion (or recession) overlaid on and tied to acertain user interface element, such as widgets, application icons,files, folders, etc., across the screen. Small user interface widgets,such as resize handles, that are often visually occluded by the fingeronce deformed may be detected and manipulated through the touchkinaesthetic feedback. In a similar example, the user may feel/explorethe edges of a digital drawing in such a way that that shapes can, forexample, be filled (i.e. virtually painted) without spilling over, evenwhen the finger occludes the edge. As another example, the display ofinteractions between a dragged object and other contents on the screenmay be felt via embodiments of the invention. For example, a moving linemay be felt through a raised icon as it is moved over a window's edge.In a similar example, while the user is dragging an object, the objectmay hit a barrier and cannot be moved any further. Embodiments of theinvention may be used with text manipulation. For example, overlayingdeformation features on top of a piece of text may facilitate textmanipulation and address the visual occlusion problem in textinteractions, and operations such as move, copy, and paste may beperformed by physically interacting with the deformation featureoverlaid on top of the text. The same functionality may also be usedwhen a virtual object moves on its own (e.g., under the effect ofvirtual gravity) so that the user may feel it moving in that direction.

In an embodiment, the array of haptic cells 212 may be used to simulatea physical controller in the touchscreen device 210. For example,deformation features may be raised from the flat touch surface 403 torepresent standard physical user interface controls, such as a D-Pad.The array of haptic cells 212 may not only simulate the physical shapeof the D-pad, but may also replicate the experience of the user wheninteracting with a real controller. For example, if the left side of theD-pad is pressed down, the haptic cells 212 under the user's finger maymove down while the haptic cells 212 on the right side may move up,which renders the interaction similar to that with a real D-padcontroller. In an embodiment, a raised feature that the user caninteract with may be provided to simulate a two dimensional (“2D”)joystick, an interactive game console, or track point found on laptopcomputers, for example.

In an embodiment, the array of haptic cells 212 may be used to simulatephysical textures, such as textures found in real life, at the macroscale. Small deformations of the haptic cells 212 may be used to producetextures ranging from sand to glass.

In an embodiment, the array of haptic cells 212 may be used to enablenon-visual user interface interactions and gestures by providingdeformation to facilitate gestures performed without visual examinationand also to provide haptic feedback. For example, finger guidance may befacilitated by creating pathways on the touchscreen 210 that guideuser's finger towards a certain target location, and then block thefinger when the finger reaches the target spot. More generally,embodiments of the invention may allow the user to “snap” to a certaingrid or shape. For example, in a drawing application, such functionalitymay create the trace of a shape, but at the same time allow the user tonot follow the shape if desired. In an embodiment in which the userinterface device 200 is a phone, the user may answer a call byinteracting with the phone in his/her pocket through inspection of adeformation feature that represents a target functionality, such asanswer the phone, hang up the phone, transfer to a wireless headset,etc. Similarly, predefined messages may be sent without having to lookat a touchscreen by being able to feel simulated tangible features ofthe touchscreen for guidance.

The programmable deformation display described above may also be used toenable a wider range of novel gesture interactions, which may lead to aricher gesture language. For example, the user may enter spatialinformation by manipulating the touch surface 403 as if the flexiblelayer 402 was made of clay or another similar material. In anembodiment, the user may manipulate/alter the elevation in a map,brightness in an image (either locally on an isolated portion of theimage or globally all over the image), may create distortions in anaudio signal by manipulating parameters mapped to a two-dimensionalsurface, or may create a mark in the simulated clay to mark a certaincontent, such as a document.

Embodiments of the invention may also be used to haptically augment freespace gestures with deformation. For example, in an embodiment, the userinterface device 200 may be moved in six degrees of freedom, anddeformation haptic effects representing various pieces of informationmay be played. For example, the user may swing the user interface device200 from left to right or rotate the user interface device 200 atdiscrete angles, and a bump haptic effect may be played or a deformationhaptic texture may be displayed via the touchscreen device 210. In anembodiment, the interaction with objects bigger than the user interfacedevice 200 may be enabled using the array of haptic cells 212. Thedeformation on the touchscreen device 210 may give the user the abilityto explore or feel a virtual object by receiving haptic informationabout only parts of the object (depending on the spatial location andthe grasping orientation, the information displayed might be different).Similar to the exploration of a sphere with a hand, the array of hapticcells 212 may display the shape of a virtual object, which may be feltby the hand at the specific locations as if the user was touching it. Inan embodiment, other properties such as softness and temperature may bedisplayed by the touchscreen device 210.

In certain gestural expressions/communications, the location, dynamics,and motion patterns of the deformation features (relief/recession) doesnot have to be exactly the same as those of the user's finger duringinteraction. For example, in an object manipulation scenario, the usermay select the object (and as a result a protrusion is created over theobject), and then tap on another point on the touchscreen device 210,where a second protrusion may be displayed. The user may then chooseeither to use the collocated protrusion (the one on top of the object),or the un-collocated protrusion to push the object across the screen. Inanother example, instead of pushing on a single protrusion that hasemerged on top of the image of the object, the user may place his/herfinger between two or more protrusions located along the boundary linesof the image and move the protrusions around with the object.

In an embodiment, the array of haptic cells 212 may display a variety ofcontent/meta data through at least one programmable deformable “button”.Such a physical button may adopt various geometries, form factors,travel strokes and haptic force profiles, thereby allowing forcustomized button interaction and confirmation experiences. The specificforce profile (resistance, dynamic inertia, detents, etc.) or clickprofile (in single press) of the button may be used to convey a varietyof meta data to the user interacting with the button. Examples ofsimilar applications include: a blinking button indicating a requiredpending action, a button that cannot be pressed down (or is recessed)unless it is active, or a button showing affordance or interactionthemes with its shape. For example, the slanted shape of a button mayindicate that the button should be pushed sideways. In an embodiment, amulti-function button may be configured to change its shape to indicatea new functionality. For example, as the user presses down on a flatbutton, the button may become rounded to show that the button may now beused as a slider. The user can then slide the button along the surface.

In an embodiment, content information may be displayed through atouchscreen full keyboard, described above, using the array of hapticcells 212. For example, haptic key magnification with proximity sensingmay be provided so that when a proximity sensor at or in the flexiblelayer 402 senses motion of a finger approaching a certain key on thetouchscreen device 210, only the key/set of keys close to the finger'starget point may be raised (i.e. haptically displayed with deformation).In an embodiment, only the keys that are most likely to be hit next,based on text prediction, may be physically deformed. The programmabledeformation may allow key edges to be made harder or softer, or buttonshapes or compliances to be changed, depending on the user's preference.In an embodiment, the keys that have a useful function in the currentuser interface may be raised, while the keys that do not have animmediate useful function or are inactive may be recessed.

In an embodiment, the visual content displayed on by the display 220 maybe augmented with certain sensory or meta data, where theadditional/augmenting information is displayed in the form ofdeformation of the touchscreen device 210 superimposed on top of theimage displayed by the display 220. For example, such additionalinformation may be the image's “visual” characteristics, such ascontrast, brightness, full spectrum photography, etc.

In an embodiment, “non-visual” information in the form of deformationfeatures (e.g., protrusion, recession, etc.) may be used to augmentcertain characters (e.g., in a movie, game, etc.) in the virtualenvironment. For example, in a gaming application, the game's characterequipment, health status, or the location and size of troops deployed ona map may be augmented (highlighted) with deformation. Such deformationfeatures, in addition to associating certain tactile sensations to thetarget character(s), may alter its (their) visual appearance(s) as well.The latter may occur if, for example, the array of haptic cells 212 forma concave or convex shape on top of the character and thus locallychange the effective optical properties of the screen. Using thismethod, it may be possible to have some areas (characters) in thevirtual environment appear blurry/diminished, while some others appearbrighter/enhanced. In an embodiment, certain areas of the user interfacedevice 200 may be used to display different audio data/sound to the userby generated isolated vibrations with the array of haptic cells 212across the touchscreen device 210.

In an embodiment, Braille language features for the visually impairedmay be displayed by, for example, turning the entire touchscreen device210 into a Braille display. Each Braille cell, representing a character,may be made up of an array of 2 by 4 dots, arranged in a rectangle-likeform factor, and users may select the size of the Braille cells, orother properties. In an embodiment, “tactile graphics” for the visuallyimpaired may be provided with the touchscreen device 210.

In an embodiment, deformation-based haptics via the array of hapticcells 212 may be used to display information to the user in a non-visualmanner, thereby minimizing the need to visually explore the touchscreendevice 210. For example, the results of a search task (text, webpage,etc.) may be displayed using relief/recession features. The searchresults may be physically “highlighted” through relief/recession,thereby allowing the user to locate them by simply examining the touchsurface 403. This may be done, for example, by either exploring thescreen using fingers, or by putting the full palm of a hand on thescreen to locate the haptically highlighted item(s). In an embodiment,information such as caller ID or message sender ID may be encoded intorelief/recession features, thereby allowing the user, for example, toidentify the caller while his/her phone is still in his/her pocket. Inan embodiment, GPS route guidance information may be displayed on thetouchscreen device 210 by simulating the road, the intersections, andthe next road to turn on in the form of physical pathways, and the caritself may be represented by a moving protruded block.

In an embodiment, the array of haptic cells 212 may be used to conveycertain meta data as the user is involved in manipulation of userinterface elements or objects in a digital environment. For example,while the user is trying to manipulate virtual objects, which arephysically simulated using the array of haptic cells 212, a programmableforce display in the form of adjustable resistance against motion (e.g.,stick, slip, detent effects, etc.) during manipulation/moving of theuser interface elements may convey information about the type, size, andother properties of the object. In addition, protrusion or recessiondeformation may be used to indicate whether grasping or selection of theobject has been successful or not. For example, when a portion of textdisplayed by the display 220 is selected or highlighted, the portion ofselected/highlighted text may have a protrusion or recess deformation ontop of it via the deformation of the haptic cells 212 associated withthe location of the selected/highlighted text.

Embodiments of the invention may be used for branding application inwhich the touchscreen device 210 may be configured to display hapticeffects that may be designed, customized, and arranged in the form of ahaptic “signature” of, or meta data representing, a specific OEM. Forexample, as the user interface device 200 is turned on, a specifichaptic pattern or signature may displayed, which may include the displayof relevant static or dynamic relief/recess information of thecorresponding OEM. In an embodiment, the logo of the target OEM may bestatically displayed as well.

In an embodiment, the systems and methods described may be used forremote information display functionality. For example, the array ofhaptic cells 212 may generate large static deformation that may bevisually detected or examined or analyzed remotely, without the need forholding or touching the user interface device 200. Information such ascaller ID, clock, calendar date, etc., if displayed with a static reliefdisplay, may be accessed by simply looking at the user interface device200 even when the user interface device 200 is sitting at a fairly fardistance away from the user.

Embodiments of the invention may also be implemented on a wearabledevice, in which silent notifications in specific contexts may bedisplayed to convey certain information when the wearable device ispressed against the body. In an embodiment, a watch may communicate timeby creating dimples on its underside and pushing against the skin of thewearer's wrist. In an embodiment, a bracelet may be configured tocommunicate an alarm by generating waves and other patterns on the skinof the wearer. In an embodiment, a shoe may be configured to create adimple against the wearer's foot to indicate the direction in whichhe/she should walk.

Embodiments of the invention may also be implemented in medical devices.For example, the inside of a laparoscopic handle may have an array ofhaptic cells 212 that may display, to the surgeon, pressure sensationrelating to a measured bio-signal during the operation, such as heartrate, etc.

In an embodiment, a range of status information related to ambience,user, phone, etc., may be communicated using the array of haptic cells212. For example, while on a phone call with the touchscreen device 210in contact with a cheek or jaw, deformation-based alerts that mayindicate the battery status or service drop may be displayed on user'sskin. In addition, even with a phone in a pocket, the user may queryinformation such as number of missed calls, received messages, orupcoming events in the calendar by tapping on the touchscreen device 210and receiving feedback encoded in the form of deformation featuresprovided by the array of haptic cells 212. In an embodiment, deformationof the array of haptic cells 212 may be used to display ambientinformation, such as barometric pressure, temperature, compassdirections, etc. In an embodiment, deformation of the array of hapticcells 212 may be used for social ambience awareness and conveyinformation such as the number and/or location of people with certaincommonalities.

Embodiments of the present invention described above may restore thelost realism/tangibility of user interface elements by physicallyaugmenting the user interface components and associated interactionswith deformable haptic cells, which may enable richer, more intuitiveand less cognitively loading interactive experiences. A tangible userinterface may also lead to more efficient and less erroneousinteractions, for example, by enabling a tangible keyboard for texttyping.

One of the potential challenges in implementing deformation displays incommercial devices is the cost of high-resolution (i.e. large number ofcells per area) arrays of haptic cells. By interpolating between forceprofile/effects between cells, as described above, it may be possible tosimulate fairly fine haptic resolutions, even with a coarse array ofdeformable haptic cells.

Embodiments of the present invention described above provide systems,methods and underlying considerations for rendering and controllinghaptically augmented tangible user interfaces and content/informationdisplays. The above-described embodiments provide various control andrendering schemes, taking user input and haptic output intoconsideration, designed around implementing an array of deformablehaptic cells, to effectively display different types of hapticinformation and enable intuitive tangible user interfaces. Theabove-described methods may expand the functionalities and increase theeffectiveness of the array of deformable haptic cells, such as increasedperceived resolution, etc., and remove the need for more sophisticatedand costly hardware.

Embodiments of the invention described above provide rendering andcontrol schemes that may be used to command an array of deformablehaptic cells to enable tangible user interface interactions andcontent/meta data display. Embodiments of the invention described aboveprovide systems and methods to display digital content and meta datausing an array of deformable haptic cells, and thereby enable richer,more informative, and more intuitive interactive experiences.Embodiments of the invention described above provide innovativetechniques for employing a haptically-enabled and programmable array ofdeformable haptic cells to display content/information/meta data in atangible, realistic, and efficient manner. The above-describedembodiments may also increase the range of applications of the array ofdeformable haptic cells and expand the breadth and depth of the hapticinformation that the user interface device can deliver.

Embodiments of the invention described above provide a new outputdimension based on programmable free form deformation that is capable ofdisplaying content and meta data in a more realistic, tangible, andintuitive manner. The use of deformation, as another haptic-basedcontent display dimension, expands the breadth and depth of informationdisplay medium and enriches data transfer bandwidth.

The embodiments described herein represent a number of possibleimplementations and examples and are not intended to necessarily limitthe present disclosure to any specific embodiments. Instead, variousmodifications can be made to these embodiments as would be understood byone of ordinary skill in the art. Any such modifications are intended tobe included within the spirit and scope of the present disclosure andprotected by the following claims.

What is claimed is:
 1. A user interface device comprising: a deformablelayer comprising a touch surface; a haptic cell covered by thedeformable layer and comprising a haptic output device; a first sensorpositioned to sense deformation of the flexible layer proximal to thehaptic cell, the first sensor configured to output a first sensor signalin response to sensing the deformation, the first sensor signal havingan electrical characteristic; and a processor in electricalcommunication with the first sensor and the haptic cell, the processorprogrammed to generate a haptic control signal in response to receivingthe first sensor signal and communicate the haptic control signal to thehaptic cell, the haptic control signal having an electricalcharacteristic defined at least in part on the electrical characteristicof the first sensor signal, the electrical characteristic of the hapticsignal corresponding to a predetermined amount of resistance to displayagainst deformation of the flexible layer.
 2. The user interface deviceof claim 1 further comprising: a plurality of haptic cells and aplurality of first sensors wherein each haptic cell comprises a firstsensor.
 3. The user interface device of claim 2 wherein a value of theelectrical characteristic of the first sensor signal of each haptic cellvaries with an amount of the deformation.
 4. The user interface of claim3 wherein, upon receiving the haptic signal, the haptic output device inthe plurality of haptic cells delivers a haptic effect selected from thegroup consisting essentially of: a wave, a ripple, movement of ajoystick, and movement of a slider.
 5. The user interface of claim 1wherein the electrical characteristic of the haptic signal is selectedfrom the group comprising magnitude, frequency, and duration.
 6. Theuser interface device of claim 1 further comprising: a second sensorpositioned to sense deformation of the flexible layer proximal to thehaptic cell, the second sensor configured to output a second sensorsignal in response to sensing the deformation, the second sensor signalhaving an electrical characteristic; and the processor is furtherprogrammed to generate the haptic control signal in response to thefirst sensor signal and the second sensor signal, the haptic controlsignal having an electrical characteristic defined at least in part onthe characteristic of the first sensor signal and the characteristic ofthe sensor second signal.
 7. The user interface device of claim 6wherein the first and second sensors are selected from the groupcomprising a strain gauge, force sensitive resistor, multi-touch sensor,pressure sensor, and multi-touch pressure sensor.
 8. The user interfacedevice of claim 7 wherein one of the first or second sensors is apressure sensor.
 9. The user interface device of claim 1 wherein theprocessor is further programmed to generate the haptic control signalonly upon the electrical characteristic of the first sensor signalhaving a value above a threshold value.
 10. A method comprising:outputting a first sensor signal from a first sensor in response todeformation of a deformable layer proximal to the first sensor, thefirst sensor signal having an electrical characteristic; generating ahaptic control signal in response to the first sensor signal, the hapticcontrol signal having an electrical characteristic defined at least inpart on a characteristic of the first sensor signal; delivering thehaptic control signal to the haptic output device; and in response toreceiving the haptic control device, the haptic output device resistingdeformation of the flexible layer, the amount of resistancecorresponding to the electrical characteristic of the first sensorsignal.
 11. The method of claim 10 wherein: outputting the first sensorsignal comprises outputting a plurality of first sensor signals, eachfirst sensor signal being output from a different first sensor; andgenerating the haptic control signal comprises generating the hapticcontrol signal in response to the plurality of first sensor signals. 12.The method of claim 11 further comprising: varying a value of theelectrical characteristic of at least one of the first sensor signals inresponse to changes in deformation of the deformable layer.
 13. Themethod of claim 12 further comprising: varying a value of the electricalcharacteristic of the haptic control signal in response to varying ofthe electrical characteristic of the first sensor signals.
 14. Themethod of claim 13 wherein the electrical characteristics of the firstsensor signals and the haptic control signal is selected from the groupconsisting essentially of magnitude, frequency, and duration.
 15. Themethod of claim 11 wherein the haptic actuators deliver a haptic effectselected from the group consisting essential of: a wave, a ripple,movement of a joystick, and movement of a slider.
 16. The method ofclaim 10 further comprising: outputting a second sensor signal from asecond sensor in response to deformation of the deformable layerproximal to the first sensor, the second sensor signal having anelectrical characteristic; generating the haptic control signal inresponse to the first sensor signal and the second sensor signal, theelectrical characteristic of the haptic control signal defined at leastin part on the characteristic of the first sensor signal and thecharacteristic of the second sensor signal.
 17. The method of claim 16wherein: the first sensor signal is associated with an amount and/orrate of deformation of the deformable layer; and the second sensorsignal is associated with a pressure exerted against the deformablelayer.
 18. The method of claim 10 wherein generating the haptic controlsignal occurs only upon the electrical characteristic of the firstsensor signal having a value above a threshold value.
 19. A userinterface device comprising: a deformable layer comprising a touchsurface; a plurality of haptic cells covered by the deformable layer andcomprising a haptic output device, a first sensor, and a second sensor;each first sensor positioned to sense deformation of the flexible layerproximal to the haptic cell, the first sensor configured to output afirst sensor signal in response to sensing the deformation, the firstsensor signal being associated with an amount and/or rate of deformationof the deformable layer, the first sensor signal having an electricalcharacteristic, the electrical characteristic of the first sensor signalhaving a value that varies with the deformation; each second sensorpositioned to sense deformation of the flexible layer proximal to thehaptic cell, the second sensor configured to output a second sensorsignal in response to sensing the deformation, the second sensor signalassociated with a pressure exerted against the deformable layer, thesecond pressure sensor having an electrical characteristic, theelectrical characteristic of the second sensor signal having a valuethat varies with the deformation; and a processor in electricalcommunication with the first sensor and the haptic cell, the processorprogrammed to generate a haptic control signal in response to receivingthe first and second sensor signals and communicate the haptic controlsignal to the haptic cell, the haptic control signal having anelectrical characteristic defined at least in part on thecharacteristics of the first and second sensor signals, the electricalcharacteristic of the haptic signal corresponding to a predeterminedamount of resistance to display against deformation of the flexiblelayer.